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Executive  Summary 


The  future  force  design  of  manned  ground  vehicles  (MGV)  will  place  increasing  workload 
demands  on  smaller  crews  and  require  a  high  operational  tempo.  A  significant  component  of  the 
Soldier’s  workload  will  be  to  maintain  situational  awareness  of  the  vehicle  surroundings  using 
indirect  vision.  While  the  indirect  vision  systems  are  supposed  to  facilitate  increased  situational 
awareness,  the  increase  in  the  amount  of  information  may  exceed  the  crew’s  cognitive  capacity. 
This  project  aims  to  enhance  Soldier  performance  within  this  context  by  combining  neural  and 
physiological  measures  of  perceptual  and  attentional  state  to  dynamically  adapt  the  presentation 
of  information  to  optimize  sustained  surveillance  of  the  vehicle  perimeter  with  only  minimal 
cognitive  workload.  The  goal  is  to  allow  the  Soldier  to  attend  to  relevant  battlefield  objects  of 
interest  using  new  technologies  at  the  critical  interface  between  the  Soldier  and  the  system. 

Technical  advances  intended  to  improve  situational  awareness  by  providing  more  information 
about  the  tactical  environment  place  high  demands  on  the  Soldier’s  limited-capacity  cognitive 
and  neural  systems.  Information  display  technologies  have  been  developed  that  filter  information 
to  prevent  performance  failures  due  to  information  overload.  However,  these  technologies  are 
typically  rigid  with  respect  to  changes  in  the  operator’s  physical  and  cognitive  state.  Thus,  a 
further  objective  of  the  project  described  in  this  report  is  to  develop  an  adaptive  framework  that 
adjusts  filtering  algorithms  to  optimize  human  performance  in  a  variety  of  operational  contexts. 
The  work  adopts  a  unique  approach  that  integrates  measures  of  behavior,  brain  activity,  and 
physiology  with  automated  information  processing  and  display  algorithms.  It  leverages  basic 
science  research  conducted  at  the  Institute  for  Collaborative  Biotechnologies  (ICB)  that  uses 
machine  learning  algorithms  to  detect  performance  failures  during  difficult  attentional  tasks 
based  on  brain  activity,  work  done  at  Science  Applications  International  Corporation  (SAIC) 
using  pattern  classification  algorithms  to  detect  threats  based  on  brain  activity,  and  work  done  at 
the  U.S.  Army  Research  Laboratory/Human  Research  and  Engineering  Directorate 
(ARL/HRED)  that  is  aimed  at  understanding  the  cognitive  constraints  on  performance  in  crew 
stations.  This  project  will  be  conducted  over  three  years.  In  Year  1,  which  has  been  completed, 
benchmarks  and  key  display  parameters  were  determined  for  the  performance  of  attentionally 
demanding  tasks.  During  Year  2,  the  parameters  established  in  the  first  year  will  be  instantiated 
in  more  realistic  situations  and  scenarios.  Finally,  the  work  in  Year  3  will  emphasize  the 
optimization  of  the  entire  system  to  improve  operator  performance.  This  project  reflects  an 
integrated  partnership  that  capitalizes  on  the  strengths  of  the  ICB,  SAIC,  and  ARL/HRED  co¬ 
investigators.  Critically,  this  work  has  already  helped  support  High-Definition  Cognition  in 
Operational  Environments  (HD-Cog)  Army  Technology  Objective  (ATO)  technologies. 
Moreover,  this  work  will  enhance  the  work  in  Brain-computer  Interactive  Technologies  (BCIT) 
in  the  Translational  Neuroscience  Branch  in  ARL/HRED. 


IX 


This  report  describes  the  results  of  10  separate  studies  that  were  conducted  by  the  team  in  the 
first  year. 
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1.  Introduction 


With  advances  in  display  and  sensor  technologies  and  with  increased  emphasis  on  a  smaller, 
more  mobile  fighting  force,  today’s  Soldier  must  deal  with  a  density  and  complexity  of 
information  that  was  unknown  in  the  past.  Although  the  intent  of  providing  Soldiers  with  more 
information  is  to  improve  their  situational  awareness  and  operational  performance  in  tactical 
situations,  the  increased  informational  content  places  high  demands  on  limited  capacity  cognitive 
and  neural  systems.  Various  automated  filtering  algorithms  and  adaptive  displays  have  been 
developed  to  help  reduce  the  amount  of  information  presented  to  the  Soldier,  but  these 
algorithms  are  rigid  and  do  not  adjust  based  on  the  user’s  cognitive  capacity,  strategies,  and  level 
of  stress.  As  a  result,  their  inefficacy  can  result  in  suboptimal  use.  Ultimately,  even  with  high 
performance  automated  filtering  systems,  the  burden  is  on  the  Soldier  to  act  on  the  information 
in  dynamic,  complex  environments.  Therefore,  it  is  critical  to  develop  technologies  that  will 
allow  the  integrated  human-machine  system  to  be  highly  adaptive  to  any  context. 

This  report  documents  the  results  of  the  first  year’s  work  on  a  3-year  project  to  develop  an 
approach  for  the  integration  of  measures  of  neural  activity  into  complex  multi-platform  human- 
machine  systems  that  will  provide  real-time  classification  of  cognitive  and  perceptual  states  and 
dynamic,  adaptive  adjustment  of  information  displays  to  accommodate  fluctuations  in  these 
states.  The  project  builds  upon  key  basic  research  conducted  at  the  Institute  for  Collaborative 
Biotechnologies  (ICB)  applying  measures  of  brain  activity  to  classify  performance  failures 
during  difficult  attentional  tasks.  The  aim  of  the  first  year  of  the  3-year  project  was  to  establish 
the  fundamental  parameters  for  optimizing  attentional  state  classification  in  dynamic  tasks  from 
measures  of  brain  activity.  These  measures  will  be  integrated  with  other  measures  of  behavioral 
performance  and  physiology  and  instantiated  in  hardware  and  software  to  monitor  and  optimize 
Soldier  performance  the  parameters.  This  work  has  already  provided  benefit  to  High-Definition 
Cognition  in  Operational  Environments  (HD-Cog)  Army  Technology  Objective  (ATO)  due  to  its 
emphasis  on  an  integrated  framework  that  takes  into  account  both  the  automated  systems  that 
Soldiers  use,  as  well  as  measures  of  performance,  neural  activity,  and  physiology.  Additionally 
this  work  will  help  support  Brain-computer  Interactive  Technologies  (BCIT)  within  the 
Translational  Neuroscience  Branch  of  U.S.  Army  Research  Laboratory/Human  Research  and 
Engineering  Directorate  (ARL/HRED). 

The  work  during  Year  1  had  four  research  objectives  supported  by  10  separate  research  studies. 
These  objectives  include: 

1 .  Establish  the  basic  parameters  that  optimize  performance  of  a  system  using  Rapid  Serial 
Visual  Presentation  (RSVP)  to  display  images  for  an  operator.  Consider  both  behavioral 
performance  (the  ability  of  the  operator  to  detect  and  report  target  threats)  and  the 
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performance  of  an  automated  algorithm  that  classifies  the  images  from  the  neural  response 
of  the  operator. 

2.  Examine  performance  when  RSVP  is  carried  out  in  conjunction  with  a  second  task. 

3.  Develop  methods  for  classifying  attentional  impairments  from  neural  data. 

4.  Build  and  test  an  application  (based  on  the  SAIC  real-time  neural  processing  system)  for 
displaying  images  and  capturing  an  operator’s  neural  and  behavioral  responses  at  the 
ARL/HRED  facility. 

This  report  begins  with  a  discussion  of  the  background  for  the  technical  approach.  It  then 
describes  the  tasks  and  deliverables  and  follows  with  a  summary  of  the  work.  Finally,  it  covers 
each  of  the  10  studies  and  ends  with  a  summary  of  the  year’s  results. 


2.  Technical  Approach 


This  6.2  research  project  will  transition  basic  research  previously  carried  out  by  the  ICB 
teammates  on  the  classification  of  human  performance  failures  during  difficult  tasks  based  on 
measures  of  brain  activity.  Within  the  past  decade,  pattern  classification  algorithms,  or  neural 
decoding  techniques,  have  been  applied  to  spatial  patterns  of  activity  measured  with  functional 
magnetic  resonance  imaging  (fMRI)  to  classify  the  types  of  objects  presented  to  and  the 
perceptual  states  of  observers  (Haynes  and  Rees,  2005;  Kamitani  and  Tong,  2005).  While  these 
neural  decoding  techniques  applied  to  fMRI  data  provide  important  clues  about  the  neuronal 
representation  of  information,  application  of  fMRI  measurements  within  a  real-time  field  context 
is  currently  not  feasible.  The  basic  research  conducted  by  ICB  co-investigators  Giesbrecht  and 
Eckstein  applies  these  classification  techniques  to  high  temporal  resolution  patterns  of  neural 
activity  acquired  with  electroencephalography  (EEG).  While  there  are  several  reports  of  stimulus 
classification  based  on  EEG  measurements  in  the  literature  (Philiastides  and  Sajda,  2006a;  Luo 
and  Sajda,  2009),  Giesbrecht  and  Eckstein  have  performed  successful  stimulus  classification 
under  difficult  dual-task  situations  that  are  potentially  relevant  to  the  battlefield  context 
(Giesbrecht  et  al.,  2009;  Ristic  et  al.,  2009).  In  this  dual-task  paradigm,  two  visual  targets  (each 
backward  masked  by  a  pattern)  are  presented  in  rapid  succession  (see  figure  la).  The  first  task 
(Tl)  required  the  discrimination  of  the  direction  of  a  central  arrow  that  was  flanked  by  arrows 
pointing  in  the  same  direction  (easy)  or  in  a  different  direction  (hard).  The  second  task  (T2) 
involved  discriminating  whether  T2  was  related  or  unrelated  to  a  context  word  presented  at  the 
very  beginning  of  the  trial.  Typically  in  this  dual  task  paradigm,  correct  identification  of  the  first 
(Tl)  leads  to  impaired  identification  of  the  second  (T2);  a  phenomenon  known  as  the  ‘attentional 
blink’  (Attentional  Blink  [AB],  figure  lb). 
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Figure  1.  Dual-task  paradigm  and  behavioral  results.  Panel  a:  Sample  stimulus  sequence.  Each  trial  began  with  the 
presentation  of  a  context  word  (1000  ms),  followed  by  the  presentation  of  the  first  target  stimulus  (80  ms), 
a  visual  mask,  and  the  second  target  word  (67  ms),  and  a  visual  mask.  The  temporal  lag  between  the  two 
targets  (T1-T2  Lag)  was  either  320  or  920  ms.  The  first  target  task  was  to  indicate  the  direction  of  the 
central  arrow  that  was  flanked  by  arrows  pointing  in  the  same  direction  (easy/low  task  load)  or  different 
directions  (hard/high  task  load).  The  second  target  task  was  to  indicate  whether  the  target  word  was 
related  or  unrelated  to  the  context  word.  Panel  b:  Behavioral  performance  on  the  second  target  task 
averaged  across  the  sample  (12  participants). 

We  applied  a  linear  pattern  classifier  (i.e.,  linear  discriminant  analysis)  to  EEG  data  evoked  by 
the  T2  and  evaluated  performance  of  the  classifier  for  separate  post-T2  time  points  using  a  k-fold 
cross-validation  scheme.  Target  amplitude  was  the  primary  classification  feature.  This  analysis 
revealed  several  key  findings.  First,  when  the  classifier  was  trained  and  tested  on  activity  evoked 
by  the  second  target,  accuracy  was  above  chance  when  the  two  targets  were  separated  by  enough 
time  to  be  outside  the  typical  AB  time  window  (figure  2).  This  was  true  regardless  of  the 
difficulty  (load)  of  the  first  target  task.  Second,  during  the  typical  AB  time  window  when 
behavioral  performance  on  the  second  target  task  is  most  impaired,  the  classifier  was  accurate 
only  when  Tl-task  load  was  low.  Because  the  AB  represents  a  failure  of  attention,  these  results 
indicate  that  patterns  of  neural  activity  recorded  during  periods  when  attentional  capacity  is 
pushed  to  its  limits  can  be  used  to  discriminate  the  information  presented  to  the  observer 
independent  of  behavior. 
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Figure  2.  Performance  of  the  classifier  in 

discriminating  whether  the  T2  word  was 
related  or  unrelated  to  the  context  word. 

Classifier  performance  was  quantified  using 
a  nonparametric  measure  of  the  area  under 
the  Receiver  Operator  Curve  (Az).  Separate 
classifiers  were  trained  and  tested  on  T2-evoked 
neural  activity  for  each  participant  measured  at 
central  and  parietal  midline  electrodes  (Cz  and  Pz). 

Another  unique  aspect  of  the  ICB  basic  research,  and  perhaps  the  most  critical  one  for  the 
present  work,  is  that  various  metrics  obtained  from  the  temporal  patterns  of  EEG  activity  can  be 
used  to  predict  the  observers’  performance  (Eckstein  et  al.,  2008;  Das  et  al.,  2009;  Giesbrecht  et 
al.,  2009).  For  instance,  in  the  dual  task  paradigm  described  above,  the  classifier  could  accurately 
discriminate  when  the  observers  made  an  error  during  the  AB  based  not  only  on  activity  evoked 
by  the  second  target,  but  also  by  activity  evoked  by  the  first  target  (figure  3).  In  other  words,  the 
classifier  could  discriminate  a  performance  failure  on  the  second  target  before  the  second  target 
was  presented  to  the  observer.  Interestingly,  the  accuracy  of  classification  errors  was  not 
constrained  by  task  load. 
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Figure  3.  Classifier  accuracy  in  discriminating  attentional  failures  based  on  inputs  derived 
from  activity  evoked  during  two  periods  after  the  second  target,  activity  evoked 
the  first  target,  and  a  Gaussian  noise  control. 

The  results  of  this  ICB  research  suggest  that  pattern  classification  algorithms  may  be  used  to 
assess  the  attentional  state  of  the  Soldier  on  a  moment-by-moment  basis,  thereby  permitting 
performance  to  be  optimized.  Using  classification  algorithms  to  detect  variations  in  attentional 
performance  is  the  key  technical  solution  that  will  be  transitioned  in  the  current  6.2  project.  The 
application  of  classification  algorithms  to  EEG  is  an  important  step  towards  real-time  monitoring 
of  the  environment  and  crew  station  operator’s  perceptual  and  cognitive  states  and  it  is  a  natural 
point  of  collaboration  between  ICB  faculty  and  current  work  being  done  by  SAIC  in  more 
complex  military  contexts,  and  the  ARL-led  BCITs. 

2.1  SAIC  Applied  and  Transitional  Research 

In  contrast  to  the  basic  research  developed  at  ICB,  SAIC’s  applied  research  and  neurotechnology 
is  based  on  established  scientific  observations  describing  how  the  human  visual  system  processes 
and  classifies  complex  imagery.  For  over  a  decade  it  has  been  known  that  the  visual  cortex  can 
perform  complex  categorization  of  visual  stimuli  within  a  few  hundred  milliseconds  of  the 
stimulus  presentation  (Thorpe  et  al.,  1996).  In  the  original  laboratory  experiment,  subjects  were 
asked  to  determine  if  a  photograph,  presented  for  only  a  few  milliseconds,  contained  an  animal. 
Using  EEG,  researchers  were  able  detect  an  animal/no-animal  categorization  signal  emanating 
from  the  frontal  and  parietal  cortex  far  earlier  (150  ms)  than  any  behavioral  response  could  be 
initiated.  Since  the  initial  finding,  various  groups  have  expanded  this  work  by  having  subjects 
perform  more  complex  categorizations  (Tanaka  and  Curran,  2001;  Tanaka  et  al.,  2005;  Scott  et 
al.,  2008).  This  idea  has  also  been  extended  to  incorporate  sequentially  presented  stimuli,  known 
as  the  RSVP  paradigm,  for  both  still  and  motion  imagery  (Luo  and  Sajda,  2009). 
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S AIC’s  recent  work  has  focused  on  transitioning  this  laboratory-based  EEG  research  into  a  more 
applied  context.  Scientific  studies  that  classify  the  perceptual  state,  based  on  EEG  signals, 
require  only  statistical  significance  to  demonstrate  their  validity.  However,  if  this  technology  is 
to  be  utilized  in  an  application,  a  much  higher  level  of  performance  is  required.  Under  the 
Defense  Advanced  Research  Project  Agency’s  (DARPA)’s  Neurotechnology  for  Intelligence 
Analysts  (NIA)  program,  SAIC  developed  EEG  classification  technology  whose  performance  far 
exceeded  any  previously  published  result.  In  this  project,  Imagery  Analysts  (IAs)  were  asked  to 
identify  targets  of  military  significance  in  satellite  imagery  (Curran  et  al.,  2009).  Using  EEG 
classification,  the  SAIC  algorithm  was  able  to  determine  when  the  IA  had  seen  a  target  only 
120  ms  after  the  image  was  presented.  The  classification  accuracy  exceeded  0.95  area  under  the 
Receiver  Operating  Characteristic  (ROC)  curve  for  many  subjects. 

Laboratory  studies  that  utilize  EEG  to  measure  and  classify  perceptual  states  often  use  repeated 
trials  to  improve  the  signal-to-noise  ratio  and  subsequent  results.  The  underlying  neural 
signatures  associated  with  a  perceptual  state  can  often  be  clearly  shown  by  averaging  over 
repeated  trials.  However,  for  many  applications  of  this  technology  it  is  critical  to  minimize  the 
number  of  required  stimulus  presentations  and  to  rapidly  process  the  EEG  signals.  Again,  under 
a  DARPA  program  (Cognitive  Technology  Threat  Warning  System-[CT2WS]),  SAIC  developed 
a  real-time  system  that  processes  the  perceptual  state  of  the  subject  on  a  trial  by  trial  basis.  Here 
subjects  were  asked  to  detect  potential  threats  (vehicles,  people,  etc.)  over  a  wide  field-of-view. 
Regions  with  potential  threats  (determined  by  an  automated  detection  algorithm)  were  presented 
to  the  subject  as  a  sequence  of  short  video  clips  (figure  4).  Using  only  the  subject’s  EEG  signals, 
the  real-time  system  identified  which  regions  contained  a  potential  threat.  Each  video  clip  was 
500  ms  long  and  was  embedded  in  a  sequence  containing  hundreds  of  such  clips.  Thus,  subjects 
were  able  to  screen  thousands  of  clips  in  a  matter  of  minutes,  with  EEG  classification 
performance  remaining  high  (figure  4).  This  real-time  system  was  tested  with  live  video  from  a 
high  resolution  imager  in  an  operational  environment  at  Yuma  Proving  Grounds  in  February, 
2009. 

The  CT2WS  real-time  system,  shown  in  figure  5,  demonstrated  the  feasibility  of  implementing 
EEG  classification  technology  in  operational  environments.  First,  it  overcame  some  of  the 
technical  challenges  inherent  in  acquiring  and  processing  EEG.  Using  automated  artifact 
detection  algorithms,  the  real-time  system  could  dynamically  redisplay  stimuli  that  were 
presented  during  periods  of  degraded  neural  signal  which  occur  with  subject  movement  or  eye 
blinks.  Second,  it  achieved  a  high  level  of  performance  in  a  stressful  operational  environment 
with  live  targets  (i.e.,  live  video  feed)  and  distracting  (i.e.,  crowded  and  noisy)  surroundings. 

The  field  test  revealed  a  critical  need  to  integrate  the  detection  of  more  subtle  attentional  states 
into  these  systems.  During  the  field  test,  subjects  were  required  to  switch  between  passively 
viewing  the  RSVP  sequence  on  one  screen  and  actively  identifying  targets,  via  a  button  press,  on 
another  screen.  Often,  during  demanding  portions  of  the  test,  classification  performance  would 
degrade.  We  hypothesized  that  the  decreased  performance  was  due  to  the  attentional  demands  of 
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task  switching  (Rogers  and  Monsell,  1995;  Slagter  et  al.,  2006).  Our  system,  however,  only 
classified  the  perceptual  state  of  the  subject  without  accounting  for  attentional  focus  or 
attentional  load.  Any  future  application  of  this  system  in  a  complex  task  environment,  such  as  an 
MGV  crew  station,  would  be  greatly  enhanced  by  attentional  measures.  By  incorporating  the 
ICB  research,  a  system  for  processing  large  amounts  of  imagery  can  be  integrated  into  the  type 
of  multi-tasking  environment  found  in  MGVs. 


Figure  4.  CT2WS  paradigm  and  results.  Panel  a:  Stimuli  consisted  of  an  RSVP  sequence  containing  short  (500  ms) 
video  clips  of  potential  targets  and  background  clutter.  Panel  b:  Video  clips  ranked  by  the  neural  response 
with  colored  border  indicating  ground-truth  (red  =  target,  blue  =  clutter);  only  the  10  highest  and  lowest 
scoring  clips  are  shown.  Panel  c:  Classifier  performance  for  one  subject  quantified  in  an  ROC  curve  (area 
under  curve  =  0.95).  Note:  imagery  from  the  CT2WS  program  has  not  yet  been  approved  for  public 
release;  images  above  are  representative  of  stimuli  used  in  the  program. 
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Figure  5.  The  SAIC  real-rime  EEG  Processing  and  Classification  System  in  use  in  the  laboratory  (left)  and  in  the 
field  (right) 

2.2  Approach  to  Research  and  Transition 

Over  a  3 -year  period,  the  current  project  will  transition  the  ICB  and  SAIC  work  from  basic 
science  and  proof  of  concept  (6.1)  into  applied  research  (6.2)  that  will  be  applicable  not  only  in 
the  context  of  future  force  MGV  concepts,  but  within  any  operational  context  that  requires  the 
Soldier  to  deal  with  multiple  complex  tasks  and  a  number  of  display  and  sensor  inputs.  The 
effort  will  integrate  the  ICB  finding  that  classification  techniques  can  be  used  to  assess 
attentional  state  with  RSVP  presentation  and  real-time  EEG  classification  techniques  previously 
developed  by  SAIC. 

The  key  technical  solution  motivating  the  current  project  is  the  application  of  pattern 
classification  algorithms  to  neural  and  physiological  data  to  determine  an  observer’s  likelihood 
of  making  an  attentional  performance  failure.  The  ICB  basic  research  serves  as  the  foundation 
which  will  be  incorporated  in  SAIC’s  real-time,  adaptive,  automated  neural  processing  system, 
and  finally  instantiated  on  Tank  Automotive  Research  Development  and  Engineering  Command 
(TARDEC)  compatible  platforms.  The  feasibility  of  this  solution  will  be  shown  by  addressing 
three  fundamental  scientific  issues. 

1 .  Optimizing  behavioral  performance  in  the  multi-task  setting.  The  demonstration  of  dual 
and  multi-task  performance  impairments  are  ubiquitous  in  the  scientific  literature 
(Raymond  et  al.,  1992;  Pashler,  1994;  Giesbrecht  and  Di  Lollo,  1998;  Giesbrecht  et  ah, 
2007).  The  key  question  here  is  whether  there  are  methods  of  presenting  critical 
information  for  multiple  tasks  that  will  minimize  these  impairments  so  as  to  optimize 
Soldier-system  performance. 

2.  Optimizing  the  classification  of  target  threats  based  on  neural  data  acquired  in  a  multi-task 
setting.  In  the  case  that  operator  performance  cannot  be  optimized,  the  question  is  whether 
there  are  methods  of  presenting  critical  information  for  multiple  tasks  to  ensure  high  neural 
decoding  performance  even  though  operator  performance  may  fail.  There  is  clear  evidence 
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based  on  the  work  done  at  ICB,  SAIC,  and  other  laboratories  that  specific  fluctuations  in 
voltage  measured  by  EEG,  both  those  that  are  tied  to  sensory  processing  and  those  that  are 
associated  with  later  decisions  and  conscious  perception,  can  be  used  to  classify  the 
stimulus  presented  to  an  observer  (Philiastides  and  Sajda,  2006a,  2006b;  Eckstein  et  al., 
2008;  Das  et  ah,  2009;  Giesbrecht  et  al.,  2009).  However,  it  is  unclear  how  these  signals 
(and  consequently  the  efficacy  of  these  neural  decoding  techniques)  are  affected  by 
sensory,  cognitive,  and  task  conditions. 

3.  Optimizing  the  classification  of  attentional  impairments  based  on  neural  data.  Based  on 
recent  work  done  at  ICB,  there  is  the  indication  that  neural  decoding  techniques  may  be 
effective  in  using  attentional  indices  in  predicting  performance  failures.  It  is  an  unresolved 
question  as  to  what  task  and  stimulus  constraints  along  with  what  properties  of  the  EEG 
signal  (amplitude,  phase,  power,  etc.)  are  most  predictive  of  performance  failures  and  how 
the  detection  of  these  failures  is  affected  by  task  demands.  Determining  these  constraints  is 
critical  for  assessment  of  the  feasibility  of  the  technical  solution. 


3.  Year  One  Tasks  and  Deliverables 


During  the  first  year  of  the  project,  the  team  performed  the  following  tasks  and  met  Milestones 

1.1  through  1.3.  Milestone  1.4  will  be  met  with  the  delivery  of  this  report. 

3.1  Tasks 

1 . 1  (ICB/SAIC)  Conduct  experiments  to  establish  the  basic  parameter  set  for  the  RS  VP  task 
that  optimizes  performance  in  terms  of  metrics  such  as  probability  of  detection  and  false 
alarm  rate  or  the  area  under  the  ROC  curve.  Analyze  the  neurophysiological  data  collected 
during  the  experiments  for  features  which  allow  categorization  of  attentional  state. 

1.2  (SAIC)  Incorporate  results  from  previous  ICB  6.1  work  into  the  SAIC  real-time  EEG 
processing  application  to  include  attentional  measures. 

1.3  (ARL)  Integrate  the  basic  SAIC  real-time  EEG  processing  application  with  the  task 
generation  hardware  and  software.  Use  the  SAIC  application  as  a  guide  for  implementation 
with  ARL  EEG  hardware. 

1 .4  (ICB)  Define  and  implement  a  primary  task  to  be  carried  out  with  the  secondary 
scanning/RSVP  task  so  that  subject  performance  can  be  measured  and  task  complexity  can 
be  manipulated. 

1.5  (ARL)  Incorporate  the  results  of  ICB/SAIC  research  (Task  1.1)  in  EEG  processing  module 
of  the  task  generation  hardware  and  software  in  order  to  optimize  performance  metrics 
such  as  probability  of  detection,  state  classification,  and  neural  pattern  categorization. 
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1.6  (ICB)  Conduct  experiments  to  develop  parameters  and  guidelines  for  dual  task 
applications. 

1.7  (SAIC)  Incorporate  the  results  of  Task  1.1  in  the  SAIC  real-time  application. 

1.8  (SAIC)  Integrate  the  SAIC  real-time  application  with  the  primary  task.  Establish  guidelines 
for  the  dual  task  implementation. 

1.9  (SAIC/ARL)  Define  a  set  of  performance  metrics  that  can  be  used  throughout  the  project  to 
assess  improvements  to  the  application.  These  metrics  should  be  relevant  to  the  application 
of  RSVP  in  a  dual  task  paradigm  and  support  meaningful  comparisons  with  standard 
methods  of  performing  these  tasks.  SAIC  will  conduct  tests  on  the  real-time  EEG  dual  task 
application  and  collect  baseline  performance  metrics. 

3.2  Technical  Deliverables  (Quarterly  Milestones) 

1.1  (SAIC)  Algorithms,  software,  and  documentation,  which  specify  hardware  requirements, 
constraints,  and  design  guidelines  for  the  incorporation  of  the  basic  SAIC  real-time  EEG 
processing  in  transitional  hardware  at  ARL. 

1.2  (ICB)  Report  documenting  the  experimental  results  with  performance  tradeoffs  and 
guidelines  for  the  RSVP  task  parameters  shown  in  table  1 . 

1.3  (SAIC)  Algorithms,  software,  and  documentation  that  incorporate  the  results  of  Task  1.1. 

1.4  (ICB/SAIC/ARL)  Year  1  final  report  documenting: 

1.4.1  Experimental  results,  parameters,  and  guidelines  for  the  basic  RSVP  task 
implementation  in  a  real-time  system  and  for  implementation  in  dual  task 
applications,  including  algorithms  and  software. 

1 .4.2  Performance  metrics  and  test  results  for  the  SAIC  real-time  system. 

1 .4.3  Hardware  and  software  guidelines  for  the  integration  of  EEG  processing  with  Army 
systems. 
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4.  Year  1  Overview 


The  Year  1  work  had  four  objectives: 

1 .  Establish  the  basic  parameters  that  optimize  performance  of  a  system  using  RS  VP  to 
display  images  for  an  operator.  Consider  both  behavioral  performance  (the  ability  of  the 
operator  to  detect  and  report  target  threats)  and  the  performance  of  an  automated  algorithm 
that  classifies  the  images  from  the  neural  response  of  the  operator. 

2.  Examine  performance  when  RSVP  is  carried  out  in  conjunction  with  a  second  task. 

3.  Develop  methods  for  classifying  attentional  impairments  from  neural  data. 

4.  Build  and  test  an  application  (based  on  the  SAIC  real-time  neural  processing  system)  for 
displaying  images  and  capturing  an  operator’s  neural  and  behavioral  responses  at  the 
ARL/HRED  facility. 

In  order  to  meet  these  objectives,  the  team  worked  to  build  a  collaborative  environment  in  which 
each  member  contributed  to  the  whole.  In  support  of  this  goal,  the  group  held  quarterly  meetings 
to  discuss  progress  and  exchange  ideas.  A  share  site  was  established  to  facilitate  the  exchange  of 
data,  imagery,  and  documents.  From  the  beginning  of  the  project,  biweekly  teleconferences  were 
held  to  discuss  the  work  of  each  group  during  the  two  week  period.  In  the  first  quarter  of  the 
project,  the  team  emphasized  the  convergence  of  the  various  experimental  paradigms  and 
methods  of  the  three  groups  so  that  each  group’s  results  could  be  related  within  a  common 
context.  Because  of  this  collaborative  approach,  the  Year  1  (FY10)  milestones  were  met,  with 
each  group  performing  different  but  complementary  tasks. 

The  first  year  results  are  summarized  by  objective. 

4.1  Objective  1:  Optimize  Performance  with  RSVP 

Work  on  this  objective  focused  on  the  parameters  set  out  in  table  1. 

Under  Task  1.1  of  the  SOW,  the  ICB  and  SAIC  systematically  examined  the  RSVP  parameters 
from  the  proposal.  The  ICB  conducted  a  series  of  studies  that  addressed  RSVP  rate  and  location 
in  the  visual  field  and  target  salience.  These  studies  show  that  performance  on  a  target 
recognition  task  degrades  when  the  RSVP  imagery  is  shifted  away  from  a  fixation  point  (the 
center  of  the  visual  field)  and  that  the  magnitude  of  the  performance  degradation  increases  at 
higher  presentation  rates. 

SAIC  has  a  large  collection  of  EEG  and  behavioral  data  that  was  acquired  in  the  laboratory  and 
in  field  tests  during  the  DARPA  CT2WS  project.  Using  this  data,  SAIC  focused  on  analyses  of 
parameters  that  affect  classification  performance  in  the  real-time  system,  guided  by  results  from 
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Barry  Giesbrecht’s  previous  (6.1)  research.  One  critical  parameter  was  identified  in  this  analysis: 
target  probability.  This  was  particularly  interesting  because  it  explained  in  part  why  the  SAIC 
real-time  system  behaved  differently  in  the  field  than  in  the  lab,  something  that  hadn’t  been 
previously  understood.  This  is  an  important  finding  for  the  overall  project.  In  real  world  systems, 
targets  often  appear  in  groups  which  means  a  system  must  gracefully  handle  long  periods  when 
no  targets  are  present  interspersed  with  periods  of  very  high  target  abundance. 

Table  1.  Experimental  variables  examined  in  Year  1  together  with  a  summary  of  the  results  to  date. 


Variable 

Example 

Conditions 

Purpose 

Results 

RSVP  location 

1.  Foveal 

2.  Peripheral 

If  performance  is  not  impaired  by 
a  peripheral  stream,  then  that 
might  permit  the  operator  to 
continue  with  primary  risk. 

Initial  results  show  rapid  fall  off  in 
performance  as  presentation  shifts 
away  from  fixation. 

Presentation 

Rate 

1.  Slow  (3  Hz) 

2.  Fast  (10  Hz) 

If  performance  is  unimpaired  at 
high  rates,  this  could  potentially 
increase  throughput. 

Q1  study  examined  8.5-12.5  Hz 
and  found  performance  degrades  at 
faster  speeds. 

Target 

Frequency 

1 .  Low  frequency 
(1:100) 

2.  High  frequency 
(1:10) 

Target  frequency  may  alter 
observer  performance,  but 
perhaps  not  classifier 
performance. 

This  was  identified  as  a  key 
parameter.  Performance  degrades 
with  very  high  target  frequency. 

Target  Salience 

1.  Low 

2.  High 

This  will  determine  visibility 
constraints  on  observer  and 
classifier  performance. 

Performance  is  limited  by  the 
observer’s  ability  to  see  the  target. 

Target  Size 

1.  Small  (1°  of 
visual  angle) 

2.  Large  5°  of 
visual  angle. 

This  will  determine  minimum 
size  constraints  for  peripheral 
presentation. 

Performance  is  limited  by  the 
observer’s  ability  to  see  the  target. 

Based  on  these  observations  from  the  CT2WS  data,  the  ICB  team  designed  and  carried  out  a 
study  to  systematically  examine  the  effects  of  target  probability.  The  ICB  team  showed  that  the 
optimal  range  of  target  frequency  is  from  10-25%.  When  targets  appear  more  frequently,  both 
behavioral  performance  (the  ability  of  the  subjects  to  press  a  key  when  they  see  a  target)  and  the 
performance  of  the  neural  signal  classifier  degrade.  This  confirmed  SAIC’s  CT2WS  analysis. 
Further  evaluation  by  SAIC  showed  that  the  effect  could  be  mitigated,  to  some  degree,  in  a  real¬ 
time  system  by  an  adaptive  thresholding  technique. 

4.2  Objective  2:  Understanding  Dual-Task  Performance  with  RSVP 

One  of  the  key  undertakings  for  this  year  was  to  define  and  implement  a  dual-task  paradigm  that 
mimics  the  multi-tasking  environment  which  is  common  in  operational  settings.  In  the  original 
proposal,  a  specific  primary  task  to  control  autonomous  mobility  was  defined  to  be  used  with 
RSVP  as  a  secondary  task.  The  decision  was  made  instead  to  define  an  auditory  task  that  mimics 
an  operator  monitoring  communications  while  simultaneously  viewing  images  with  RSVP. 

There  were  several  compelling  reasons  for  this  change.  Initial  results  by  the  ICB  showed 
significant  degradation  in  performance  when  the  RSVP  stream  is  presented  away  from  fixation 
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(even  para-foveally).  This  means  that  if  the  primary  task  is  also  visual,  the  operator  cannot  easily 
perform  both  tasks  simultaneously.  In  order  to  handle  both  tasks,  the  primary  task  must  be 
interrupted  periodically  to  present  the  RSVP  sequence.  In  this  type  of  implementation,  the  tasks 
are  executed  serially,  without  making  significant,  simultaneous  demands  on  the  cognitive 
resources  of  the  operator.  Since  the  goal  of  the  Year  1  work  was  to  examine  the  impact  of  RSVP 
on  a  task  with  shared  cognitive  resources,  the  team  decided  to  implement  a  non- visual  primary 
task. 

The  ICB  conducted  the  initial  cross-modal,  dual-task  study  in  the  first  quarter.  For  the  particular 
auditory  and  visual  tasks  they  used,  they  found  that  overall  performance  was  impaired, 
particularly  when  the  temporal  interval  between  the  two  tasks  was  brief.  During  the  second 
quarter,  the  ICB  and  SAIC  teams  worked  together  to  define  and  implement  a  new  set  of  auditory 
and  visual  tasks  based  on  the  SAIC  real-time  system.  Both  teams  began  running  subjects  in  dual¬ 
task  experiments.  The  ICB  team  found  that  there  was  no  degradation  in  either  behavioral  or 
classifier  performance  when  subjects  were  looking  for  visual  targets  either  as  a  single  task  or 
while  simultaneously  listening  for  a  target  word.  SAIC  had  similar  results  with  a  slightly  more 
difficult  cross-modal  paradigm.  SAIC  also  found  that  there  is  interference  between  the  auditory 
and  visual  tasks  in  the  form  of  delayed  response  times  for  the  visual  task  when  auditory  stimuli 
are  also  present. 

New  results  uncovered  in  this  research  are  that  RSVP  can  be  used  in  conjunction  with  common 
auditory  tasks,  such  as  monitoring  communications.  The  results  also  point  to  the  possibility  of 
building  a  second  classifier  for  auditory  target  recognition  that  can  be  operating  in  parallel  with 
the  RSVP  classifier  in  a  real-time  system. 

4.3  Objective  3:  Classifying  Attentional  Impairments 

By  the  third  quarter  of  the  project,  all  three  teams  had  tested  a  number  of  subjects  and  collected 
neural  and  behavioral  data  using  an  RSVP  paradigm.  To  address  the  third  objective,  the  teams 
analyzed  their  experimental  data  looking  for  features  in  the  EEG  signals  that  could  predict 
performance  errors,  that  is,  either  incorrect  responses  from  subjects  or  errors  in  classifying 
subjects’  neural  signals.  Finding  reliable,  predictive  features  could  have  a  major  impact  by 
enabling  new  types  of  robust,  adaptive  Soldier-system  interfaces.  The  ICB  team  looked  at  all 
trials  when  the  subjects  were  presented  with  a  target.  They  compared  the  instances  when  the 
subjects  responded  correctly  (“hits”)  with  those  when  they  didn’t  (“misses”).  They  found  that 
they  could  correctly  classify  these  two  categories  at  a  level  above  chance  based  upon  features  of 
the  post-stimulus  EEG  signal.  The  SAIC  team  looked  at  the  temporal  dynamics  of  the  score  from 
SAIC’s  real-time  EEG  classifier  and  found  a  feature  that  indicates  when  the  system  is 
performing  well  and  classifying  accurately.  The  ARL  team  looked  at  the  relationship  between  a 
subject’s  behavioral  reaction  time  in  an  RSVP  visual  target  detection  task  and  their  EEG  signals. 
They  found  a  significant  difference  in  the  evoked  neural  response  from  the  trials  with  the  slowest 
reaction  times  and  those  with  the  fastest. 
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4.4  Objective  4:  Integration  of  an  RSVP  Application  at  ARL 

ARL  and  SAIC  worked  together  during  the  first  quarter  of  this  year  to  duplicate  key  components 
of  the  SAIC  real-time  system  at  ARL.  SAIC  delivered  a  database  of  video  clips  from  the 
DARPA  CT2WS  program  and  supported  the  ARL  scientists  in  duplicating  the  CT2WS  RSVP 
experiments  with  that  data.  The  ARL  team  then  conducted  several  studies  with  this 
implementation.  Study  results  are  summarized  under  Objective  3. 

As  the  project  transitions  from  prototype  to  integration  with  the  crew  station  simulator,  it  is 
important  to  use  a  common  set  of  performance  metrics  to  measure  improvements  and  highlight 
problems.  The  metrics  quantify  the  performance  of  a  system  that  incorporates  RSVP  and  the 
classification  of  neural  signals  to  identify  targets  of  interest  in  imagery.  These  metrics  will  be 
used  in  the  next  two  years  to  measure  the  improvement  in  performance  as  research  results  are 
incorporated  in  the  prototype  system.  The  metrics  will  also  be  utilized  to  answer  the  fundamental 
question  of  this  research  for  different  systems:  does  the  integration  of  neural-based  processing  in 
a  given  system  significantly  improve  its  overall  performance?  The  metrics  defined  address  this 
question  in  terms  of  accuracy,  sensitivity,  and  throughput. 

Table  2  summarizes  the  effort  in  Year  1  for  each  of  the  four  objectives.  It  cross  references  these 
objectives  to  the  tasks  and  milestones. 

Table  2.  Cross-reference  objectives  to  tasks  and  milestones  and  Year  1  status. 


Objective 

Tasks 

Milestones 

Performers 

1 :  RSVP  Parameters 

1.1,  1.7 

2,4 

ICB,  SAIC,  ARL 

2:  Dual-task 

1.4,  1.6,  1.8 

4 

ICB,  SAIC 

3:  Attentional  State 

1.1,  1.2 

3,4 

ICB,  SAIC,  ARL 

4:  Transition  to  ARL 

1.3,  1.5,  1.9 

1,4 

ARL,  SAIC 

5.  Parameters  to  Optimize  Performance  with  RSVP 


To  address  the  first  objective,  the  team  conducted  four  separate  studies  that  examined  aspects  of 
Rapid  Serial  Visual  Presentation  of  images.  The  studies  were  directed  toward  how  best  to  present 
imagery  so  that  a  neural  signal  classifier  could  accurately  detect  the  viewer’s  response. 

5.1  Study  1:  RSVP  Location  and  Presentation  Rate  (ICB) 

For  this  study  (figure  6a),  participants  viewed  short  (approximately  20  items)  RSVP  streams  of 
letters.  In  half  the  streams  an  X  or  a  3  was  present  or  absent  and  at  the  end  of  each  stream, 
participants  had  to  indicate  whether  the  target  (i.e.,  an  X  or  a  3)  was  present  or  absent.  The  key 
manipulation  in  this  task  was  RSVP  rate  (8.5  Hz,  10  Hz,  and  12.5  Hz)  and  RSVP  distance  from 
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the  fixation  point  (0.36°,  0.46°,  and  0.73°).  The  mean  percentage  of  correct  responses  in  this  task 
(i.e.,  hits  and  correct  rejections)  is  shown  in  figure  6b. 

5.1.1  Results 

The  key  finding  in  this  study  is  that  there  was  an  interaction  between  RSVP  location  and 
presentation  rate,  such  that  when  the  RSVP  stream  is  presented  close  to  fixation  there  is  no  effect 
of  presentation  rate,  however,  a  slight  shift  away  from  fixation  results  in  a  large  effect 
(approximately  15%)  on  presentation  rate.  Given  that  the  stimuli  in  these  sequences  are  highly 
over  learned  (i.e.,  letters),  this  suggests  that  moving  the  RSVP  stimuli  to  the  periphery  in  the 
context  of  the  SAIC  RSVP  task  may  not  be  effective. 


Figure  6.  Panel  a:  Sample  stimulus  sequence.  Panel  b:  Mean  percent  correct  responses  as  a  function 
of  RSVP  rate  and  location. 

5.2  Study  2:  RSVP  Location  and  Target  Salience  (ICB) 

The  goal  of  this  task  was  to  investigate  the  role  of  target  salience  in  two  different  viewing 
conditions  (figure  7a).  In  one  condition,  RSVP  sequences  of  cars  were  presented  in  the  center  of 
the  screen  (central  task).  In  the  second  condition,  an  RSVP  stream  of  letters  was  presented  in  the 
center  of  the  screen  and  during  that  stream  a  face  or  a  car  was  presented  in  the  periphery,  just 
above  or  just  below  fixation.  In  each  condition,  subjects  had  to  indicate  whether  a  face  was 
present  or  absent.  The  key  manipulation  in  this  study  was  that  target  salience  was  manipulated  by 
titrating  the  level  of  noise  in  the  image  using  a  modified  staircase  procedure.  The  key  measure 
was  the  standard  deviation  of  the  noise  distribution  that  was  required  to  maintain  performance 
between  75-85%. 
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5.2.1  Results 


The  results  of  this  study  are  plotted  as  a  function  of  experimental  trial  in  figure  7b  and  averaged 
across  the  experiment  in  figure  7c.  There  are  three  key  results.  First,  overall  noise  level  is  lower 
when  the  targets  were  presented  in  the  periphery.  Second,  maximum  noise  level  peaked  at  about 
the  same  time  and  at  about  the  same  level  in  both  conditions.  Third,  the  drop  in  noise  level  was 
more  precipitous  in  the  peripheral  condition.  Together  these  results  suggest  that  similar  levels  of 
performance  may  be  obtained  in  central  and  peripheral  tasks,  performance  in  the  peripheral  task 
is  less  robust  to  targets  that  are  not  salient. 


Central  Task 


Present  Absenl 


Titrated  Noise  Level  (75-85%) 
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Figure  7.  Panel  a:  Sample  stimulus  sequences  in  the  central  and  peripheral  tasks.  Panel  b:  Mean  standard 

deviation  of  the  noise  distribution  required  to  maintain  performance  between  75%-85%  plotted  as  a 
function  of  trial.  Panel  c:  Same  as  Panel  b,  except  averaged  across  the  entire  experiment. 

5.3  Study  3 A:  Target  Frequency  (ICB) 

The  goal  of  this  study  was  to  bridge  the  gap  between  the  SAIC  RSVP  task  and  the  tasks  we  have 
been  using  in  the  ICB  6.1  work.  Here  we  are  using  the  face-car  stimuli  (Das  et  al.,  2010)  with  the 
same  timing  parameters  as  those  used  in  the  SAIC  task.  The  participant’s  task  was  to  monitor  the 
sequence  for  faces  and  to  press  the  space  bar  when  one  is  detected.  A  face  occurred  only  10%  of 
the  time.  EEG  data  from  32  electrodes  were  collected  while  subjects  performed  this  task. 
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5.3.1  Results 

Single-trial  voltages  from  the  EEG  data  were  submitted  to  a  linear  discriminant  analysis,  similar 
to  what  we  have  used  in  the  past  (i.e.,  Das  et  ah,  2010)  and  the  preliminary  results  for  five 
observers  using  one  electrode  (P04)  are  shown  in  figure  8.  There  are  three  key  findings.  First, 
classifier  accuracy  peaks  at  about  300  ms  after  the  presentation  of  the  stimulus.  This  can  be  seen 
in  both  the  single  subject  data  (Panel  A)  and  in  the  subject  mean  (Panel  B).  Second,  there  are 
clear  individual  differences  in  classifier  performance.  Third,  the  peak  classifier  performance  in 
this  task,  in  which  the  probability  of  a  face  is  0.1,  is  approximately  150  ms  later  than  in  our 
recently-published  study  in  which  the  probability  of  a  face  was  0.5  (Panel  C,  Das  et  ah,  2010). 
When  this  difference  is  considered  in  light  of  the  SAIC  work  on  target  probability  (see  Study  4), 
it  suggests  that  target  probability  is  likely  a  critical  factor  in  determining  the  spatiotemporal 
dynamics  of  classifier  performance. 


Figure  8.  Panel  a:  Classifier  percent  correct  for  each  observer  using  EEG  amplitude  at  electrode  P04  plotted  as  a 
function  of  stimulus  onset.  Panel  b:  Classifier  performance  averaged  over  subjects.  Panel  c:  Classifier 
performance  from  Das  et  al.,  (2010)  in  which  the  probability  of  a  face  as  0.5. 

5.4  Study  3B:  Target  Frequency  (ICB) 

Based  on  the  results  of  Study  3A,  this  study  tested  the  hypothesis  that  target  probability  is  likely 
a  critical  factor  in  determining  the  spatiotemporal  dynamics  of  classifier  performance  by 
conducting  a  multi-session  experiment  in  the  RSVP  task  (2  Hz)  in  which  the  target  was  a  face 
and  the  nontargets  were  cars  (both  were  embedded  in  noise).  Each  observer  participated  in  four 
sessions.  In  each  session,  only  the  target  probability  was  manipulated:  with  values  of  0.05,  0.10, 
0.25,  or  0.50  randomly  assigned  to  each  session.  A  total  of  eight  subjects  completed  four 
separate  sessions  (32  total  sessions),  where  each  session  differed  in  terms  of  the  probability  that  a 
target  was  presented. 
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5.4.1  Results 


The  behavioral  performance  (d’)  for  all  participants  is  shown  in  figure  9  as  a  function  of  target 
probability.  Clearly,  sensitivity  to  the  target  declines  as  a  function  of  increasing  target 
probability. 


0.05  0.1  0.25  0.5 

p(target) 


Figure  9.  Behavioral  performance  (cF)  plotted  as  a  function 

of  target  probability  averaged  across  all  observers  (n=8). 

Shown  in  figure  10  are  the  results  of  a  new  pattern  classification  approach,  which  identifies  the 
optimal  spatial  features  that  carry  the  most  discriminative  information  for  the  particular 
classification  problem  in  question.  The  resulting  mean  spatial  filters  are  plotted  as  a  function  of 
target  probability  in  figure  10.  Visual  inspection  of  the  spatial  filters  indicates  that  as  target 
probability  decreases  there  appears  to  be  an  increase  in  the  distribution  of  discriminative 
information  across  the  parietal  and  occipital  electrodes.  Under  high  probability  conditions  the 
most  discriminative  information  appears  to  be  largely  isolated  at  lateral  occipital  electrodes. 

^ 

0  05  0  10  0.25  0  50 


Figure  10.  The  mean  spatial  filters  depicting  the  source  of  discriminative 
information  between  targets  and  nontargets  as  a  function  of 
target  probability. 


To  demonstrate  that  the  pattern  observed  in  the  average  spatial  filters  is  not  driven  by  a  small 
number  of  subjects,  the  spatial  filters  for  each  subject  are  plotted  for  each  condition  in  figure  11. 
All  but  one  subject  (S8)  shows  the  same  trend  for  a  more  diffuse  distribution  under  low  target 
probability  and  more  focused  lateral  occipital  distribution  at  high  target  probabilities.  The  results 
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of  both  the  group  mean  and  single  subject  data  clearly  indicate  that  the  optimal  electrodes  for 
target  classification  changes  as  a  function  of  target  probability.  Thus,  if  the  target  probability  is 
known  for  a  specific  task  context,  then  the  classification  algorithm  could  be  optimized  to 
emphasize  the  appropriate  electrodes  for  that  condition. 


Figure  11.  The  spatial  filters  for  each  individual  subject  depicting  the  source  of  discriminative  information  between 
targets  and  nontargets  as  a  function  of  target  probability  (top  -  bottom  rows  =  0.05,  0.10,  0.25,  and  0.5, 
respectively). 

After  the  spatial  filters  were  constructed,  target  classification  was  performed.  We  have  adopted  a 
variant  of  the  linear  discriminant  analysis  (LDA)  approach  to  classification  that  is  based  on 
Bayes  Theorem.  The  results  shown  in  figure  12  are  focused  on  the  classification  of  the  stimulus 
using  the  first  500  ms  of  stimulus-evoked  EEG  activity.  Each  figure  shows  the  ROC  curves  for 
each  subject  (colored  lines)  and  group  average  (black  line)  and  associated  Area  Under  the  Curve 
(AUC)  in  each  target  probability  condition.  There  are  two  interesting  results  from  the  ROC 
analysis.  First,  the  overall  (AUC)  is  best  when  the  target  probability  is  0.10  or  0.25.  Second, 
there  appears  to  be  a  large  amount  of  variability  across  individuals.  The  source  of  these 
individual  differences  is  interesting  both  from  a  basic  science  and  applied  science  perspective 
and  it  is  something  that  we  are  currently  investigating  further  with  subsequent  analyses. 
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Figure  12.  Pattern  classification  ROC  curves  plotted  as  a  function  target  probability.  Colored  lines  represent  each 
subject,  the  black  line  the  group  mean.  Panels  a-d  show  the  results  for  each  target  probability  condition 
0.05,  0.10,  0.25,  and  0.50  (Cl-4,  respectively).  TPR=True  Positive  Rate,  FPR=False  Positive  Rate. 

Because  target  probabilities  vary  across  task  contexts,  it  is  important  to  know  how  robust  a 
classification  model  is  with  respect  to  changes  in  target  probability.  To  investigate  this  issue,  we 
ran  several  analyses  where  the  pattern  classification  algorithm  was  trained  on  one  target 
probability  condition  and  tested  on  the  remaining  three  conditions.  The  ROC  curves  for  each  of 
these  analyses  are  shown  in  figure  13.  There  are  two  interesting  results.  First,  there  is  an  overall 
reduction  in  classification  AUC  for  all  training  conditions.  Second,  the  best  performance  was 
obtained  when  the  classification  algorithm  was  trained  on  the  0.5  probability  condition.  This 
contrasts  the  previous  analysis  which  trained  and  tested  on  the  same  condition  and  suggests  that 
when  a  target  probability  is  unknown  that  using  the  higher  probability  condition  may  be  the  best 
option. 


Figure  13.  Similar  to  figure  12,  except  when  training  on  one  condition  and  testing  on  the  remaining  conditions. 

5.5  Study  4:  Target  Frequency  (SAIC) 

For  this  study,  SAIC  leveraged  an  extensive  dataset  collected  during  the  CT2WS  project,  both  in 
the  laboratory  and  in  field  tests,  to  focus  on  parameters  that  affect  classification  performance  in 
the  SAIC  real-time  system.  This  was  of  particular  interest  since  we  observed  a  decrease  in 
performance  for  the  same  subjects  in  the  field  test  as  compared  with  the  laboratory.  The  results 
of  neural  signal  classification  were  significantly  less  accurate  in  the  field.  One  factor  that  quickly 
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emerged  was  the  difference  in  target  probability.  In  the  lab,  target  probability  was  pre-selected 
and  fixed.  In  the  field  sessions,  target  probability  varied  greatly. 


Figure  14  illustrates  the  difference  in  target  probability  between  a  lab  and  field  experiment.  The 
first  two  rows  show  the  classifier  scores  and  average  target  event  related  potential  (ERP)  for  a 
laboratory  experiment.  The  second  two  rows  show  the  same  data  from  the  field  experiment. 

There  was  a  clear  difference  in  target  probability  (i.e.,  the  ratio  of  red  diamonds  to  black  circles 
per  unit  of  time)  between  these  experimental  sessions.  While  the  lab  session  had  a  constant  target 
probability  the  field  session  varied  form  very  low  to  very  high  target  probability.  The  resulting 
effect  on  the  ERP  is  clear,  showing  a  significant  degradation  in  field  ERP. 


OOOQQOOO 


Figure  14.  Automated  neural  processing  system  comparison.  Top  rows  represent  one 
experimental  session,  bottom  rows  represent  one  field  session.  Scatter  plots 
show  the  classifier  score  for  each  stimuli  as  a  function  of  time 
(NTG  =  non-target,  TRG  =  target).  Bad  events  are  when  the  EEG  could  not 
be  classified  due  to  artifact  or  noise.  The  topographic  plots  are  average  target 
minus  non-target  ERP  over  the  entire  session. 


5.5.1  Results 

If  we  focus  on  the  Field  scatter  plot  in  figure  14,  it  is  clear  that  when  the  target  probability  was 
high,  during  the  second  half  of  the  test,  the  target  scores  were  low  (below  the  threshold  of  0)  and 
thus,  many  targets  were  misclassified  by  the  system  as  nontargets.  In  general,  as  target 
probability  increased  the  average  classifier  score  decreased,  for  both  targets  and  non-targets 
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(figure  15).  This  resulted  in  a  decrement  in  overall  classifier  performance  for  those  sessions. 
However,  with  knowledge  of  target  probability,  the  classifier  score  can  be  adjusted  to  offset  the 
decrease.  Figure  16  illustrates  how  correcting  for  target  probably  increases  performance.  Of 
course,  an  operational  system  will  not  have  access  to  true  target  probability;  however,  target 
probability  can  be  predicted  directly  from  the  score  distribution.  Specifically,  target  probability  is 
inversely  related  to  the  kurtosis  of  the  scores.  A  system  that  continuously  monitors  recent 
classifier  scores  would  likely  be  able  to  estimate  target  probability. 


Figure  15.  Classifier  Score  and  Target  Probability.  Average  score, 
calculated  over  2-min  epochs,  plotted  as  a  function 
of  target  probability  within  that  epoch.  TRG 
scores  in  red,  mean  NTG  scores  in  black.  R 
values  indicate  correlation  coefficient. 


Figure  16.  Classifier  performance  (area  under  the  ROC  curve)  from  the  CT2WS  field  test.  Panel  a  shows  the 
original  results  while  Panel  b  shows  the  results  if  the  classifier  scores  are  adjusted  based  upon  target 
probability. 
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6.  Understanding  Dual  Task  Performance 


To  meet  the  second  objective,  to  examine  performance  when  RSVP  is  carried  out  in  conjunction 
with  a  second  task,  the  team  developed  a  dual  task  experiment  that  included  visual  target 
detection  with  RSVP  and  a  simultaneously  occurring  auditory  task.  SAIC  and  the  ICB  each  used 
their  respective  RSVP  tasks  and  added  a  primary  task  of  listening  for  a  call  sign.  For  the  ICB,  the 
visual  targets  were  faces  or  cars.  SAIC  used  the  video  imagery  from  CT2WS.  The  motivation  for 
this  specific  dual-task  paradigm  was  twofold.  First,  both  the  visual  and  auditory  task  are  akin  to 
something  a  Soldier  would  be  doing  while  interacting  with  a  crew-station  system.  Second, 
evidence  suggests  that  performance  degradation  from  cross  modality  multitasking  is  much  less 
than  multitasking  in  the  same  modality  (Alais  et  al.,  2006).  Most  all  studies  of  RSVP  have  been 
structured  so  that  the  RSVP  is  the  central  and  only  task.  Two  studies  were  carried  out  to  examine 
RSVP  with  an  auditory  task. 

6.1  Study  5:  Guidelines  for  Dual  Tasks  (ICB) 

In  this  study,  the  visual  RSVP  task  consisted  of  the  target  (a  face)  presented  with  0.10  probability 
in  a  2-Hz  RSVP  of  cars.  Additionally,  auditory  two-syllable  words  selected  from  the  military 
alphabet,  presented  using  text  to  speech  functions,  are  played  through  speakers  simultaneously 
with  every  sixth  image.  When  performing  the  word  detection  task,  a  target  word  was  randomly 
selected  at  the  beginning  of  that  session  that  occurred  with  a  0.10  probability.  Participants 
performed  three  different  conditions:  face  detection  only,  word  detection  only,  or  simultaneous 
dual  detection.  Order  was  randomized,  but  restricted  so  that  the  word  detection  only  condition 
followed  either  face  detection  only  or  dual  task,  thus  ensuring  that  participants  were  practiced  in 
face  detection  before  performing  the  word  task  in  isolation.  This  task  was  designed  to  mimic  a 
task  Soldiers  may  do  when  monitoring  auditory  communications  for  their  call  sign. 

6.1.1  Results 

Seventeen  participants  completed  this  study.  Overall  behavioral  performance  is  shown  in  figure 
17.  The  only  significant  effect  on  performance  was  the  difference  between  the  auditory  and 
visual  tasks.  There  was  no  significant  dual-task  impairment. 
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Figure  17.  Behavioral  performance  (percent  hits  and  percent  false  alarms) 
plotted  as  a  function  of  the  number  of  tasks  and  task  type. 


The  results  of  the  pattern  classification  analysis  using  linear  discriminant  analysis  are  shown  in 
figure  1 8.  It  is  important  to  note  that  this  analysis  is  focused  on  using  neural  activity  to  predict 
whether  a  face  was  present  in  the  visual  stream  in  the  conditions  in  which  the  observers  were 
performing  on  the  auditory  task  alone  (auditory  single),  both  tasks  (visual  dual),  or  the  visual 
task  alone  (visual  single).  There  are  two  key  results  that  emerged  from  this  analysis.  First,  the 
performance  of  the  classifier  was  unaffected  whether  performing  the  visual  task  alone  or  while 
performing  both  the  auditory  and  visual  task.  Second,  performance  of  the  classifier  is  severely 
impaired  when  subjects  are  instructed  to  perform  the  auditory  task  only.  These  results  suggest 
that  the  real  time  classifier  may  only  be  successful  under  multi-task  conditions  when  observers 
are  told  to  actively  attend  to  the  visual  task. 


Classifying  Visual  Target  (Face)  vs.  Nontarget  (Car) 


Auditory  Single  Task 
Visual  Single  Task 
Dual  Task 


Time  (ms) 


Figure  18.  Pattern  classifier  performance  decoding  the  visual  stimulus 
plotted  as  a  function  of  time  and  task  condition. 
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In  addition  to  the  pattern  classification  analysis  described  above,  we  have  also  performed  a  series 
of  correlations  between  behavioral  performance  and  classifier  performance.  This  analysis 
revealed  significant  positive  correlations  between  classifier  performance  and  behavioral 
performance  in  both  the  single  and  dual  task  conditions  (figure  19:  Panels  A  and  B).  Moreover,  a 
correlation  between  the  classification  performance  in  the  single  task  condition  and  classification 
performance  in  the  dual  task  condition  was  also  significant  (figure  19:  Panel  C).  These  results 
suggest  that  the  classifier  can  be  used  as  a  proxy  for  human  performance  in  different  task 
conditions  and  that  the  classifier  in  one  task  context  can  serve  as  a  performance  proxy  for 
classifier  performance  in  a  different  task  context. 


A.  Single  Task  Visual  Target 
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S.  Dual  Task  Visual  Target 
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C.  Correlating  Classifiers 


Figure  19.  Correlations  between  behavioral  performance  and  classifier  performance  in  the 
single  (Panel  a)  and  dual  (Panel  b)  task  conditions  and  between  single  and 
dual  task  classifier  performance  (Panel  c). 

In  addition  to  this  analysis,  a  second  type  of  classifier  was  applied  and  the  results  analyzed.  The 
second  classifier  was  the  LDA  classifier  described  in  the  previous  study.  This  first  set  of  analyses 
focused  on  classification  of  the  visual  stimulus  under  single  and  dual  task  conditions  and 
classification  of  the  auditory  stimulus  under  single  and  dual  task  condition.  The  results  of  this 
analysis  are  plotted  in  figure  20  and  in  table  3.  Interestingly,  the  AUC  when  classifying  the 
visual  stimulus  is  unaffected  by  the  dual  task  manipulation,  but  the  AUC  classifying  the  auditory 
stimulus  is  impaired  under  dual  task  conditions  relative  to  single  task  conditions. 
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(a)  Single-  task  -  Visual  rafget 


(b)  Dual  risk  -Visual  target 


(c)  Single  task  -  Audilory  target 


(d)  Dual  task  -  Auditory  larger 


Figure  20.  Pattern  classifier  ROC  curves  when  decoding  the  visual  and  auditory  stimuli.  Gray  lines  are  for 
individual  subjects.  The  black  line  indicates  the  group  mean. 


Table  3.  Classifier  AUC  as  a  function  stimulus  (Vis./Aud.) 
and  number  of  tasks  (Single/Dual). 


Vis.  Single 

Vis.  Dual 

Aud.  Single 

Aud.  Dual 

Mean 

0.758 

0.759 

0.809 

0.751 

SD 

0.093 

0.086 

0.073 

0.069 

Max 

0.889 

0.896 

0.906 

0.915 

Min 

0.585 

0.624 

0.666 

0.676 

A  follow-up  analysis  was  performed  training  the  classifier  on  one  task  (e.g.,  single)  and  then 
testing  it  on  another  task  (e.g.,  dual)  within  each  stimulus  modality.  In  all  cases,  classification 
was  well  above  chance,  but  significantly  degraded  when  testing  on  data  that  contained  a  different 
number  of  tasks. 

We  also  conducted  analyses  that  revealed  significant  positive  correlations  between  individual 
differences  in  classification  accuracy  under  single  task  conditions  with  individual  differences  in 
classification  accuracy  under  dual  task  conditions.  The  key  result  was  that  within  each  modality 
there  were  significant  correlations  between  single  and  dual  task  classifier  performance.  The 
visual  task  correlation  was  the  best  at  r(15)=0.89.  The  auditory  task  correlation  coefficient  was 
much  less  robust,  but  still  significant  at  r(15)=0.55.  Together  these  analyses  suggest  that 
although  there  is  degradation  in  classifier  performance  when  training  on  one  task  and  testing  on 
the  other,  one  could  still  potentially  predict  the  performance  in  one  condition  using  performance 
in  the  other  condition,  particularly  for  the  visual  task. 

6.2  Study  6:  Guidelines  for  Dual  Tasks  (SAIC) 

S  AIC’s  dual  task  was  an  extension  of  the  RSVP  paradigm  used  in  the  CT2WS  program. 
Specifically,  500-ms  video  sequences  were  presented  in  a  serial  and  uninterrupted  fashion  for  a 
number  of  2-min  blocks.  Subjects  maintained  fixation  on  the  video  stream  and  pressed  a  button  if 
they  saw  a  target  (e.g.,  person  or  vehicle).  In  addition  to  this  visual  task,  subjects  were 
simultaneously  listening  to  a  communication  channel  of  spoken  words  (military  alphabet).  They 
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were  instructed  to  push  a  second  button  if  they  heard  specific  target  words  or  call  signs  (e.g., 
“november,”  “zulu”). 

6.2.1  Results 

Behavioral  results  from  our  dual  task  experiment  indicate  that  the  majority  of  subjects  can 
perform  both  tasks  with  reasonable  accuracy.  However,  even  in  this  relatively  simple 
multitasking  scenario,  the  addition  of  the  auditory  detection  task  significantly  affects  the  reaction 
time  in  the  visual  (RSVP)  task.  Figure  21a  shows  the  relationship  between  stimulus  onset 
asynchrony  (SOA),  the  time  difference  between  the  visual  and  auditory  presentation,  and  the 
reaction  time  (RT).  Here,  the  reaction  time  is  delayed  as  the  SOA  approaches  the  average 
response  time  (about  500  ms)  for  visual  target  detection.  This  effect  is  also  reflected  in  the 
overall  reaction  time  distributions  (figure  21b);  where  the  average  reaction  time  for  trials  with 
both  visual  and  auditory  stimuli  (yellow)  is  faster  for  trials  with  visual  stimuli  alone  (blue).  This 
type  of  behavioral  interference  is  common,  even  in  the  simplest  types  of  multitasking  (Pashler, 
1994). 


Figure  21.  Behavioral  and  Classifier  Performance  for  Dual  Task  Experiment  (all  subjects).  The  task  was  to 

simultaneously  detect  targets  in  video  imagery  and  specific  words  in  an  audio  communication  stream. 
Panel  a:  The  relationship  between  SOA  and  RT.  Panel  b:  RT  distributions  for  trials  with  interference 
(visual  and  auditory  together,  yellow)  and  without  interference  (visual  alone,  blue).  Panel  c: 
Behavioral  accuracy  (ACC)  and  classifier  area-under-the-curve  (AUC)  with  and  without  interference. 
Note  that  interference  affected  RT  but  not  ACC. 
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In  addition  to  behavioral  interference,  we  found  evidence  for  interference  in  the  classification  of 
the  evoked  response  (figure  21c).  For  the  majority  of  subjects,  the  classifier  performed  equally 
well  for  trials  with  and  without  auditory  interference.  However,  several  subjects  showed  a 
significant  degradation  in  classifier  performance  for  trials  where  visual  and  auditory  stimuli  were 
presented  together.  Interestingly,  these  subjects  tended  to  have  relatively  lower  area  under  the 
ROC  curve  (AUC)  values,  even  without  auditory  interference,  indicating  poorer  classifier 
performance.  This  type  of  interference  could  be  a  consequence  of  overlap  in  the  evoked  response 
from  visual  and  auditory  stimuli.  To  illustrate  this,  figure  22  shows  that  the  scalp  topology  of  the 
auditory  (for  target  words)  and  the  visual  (for  target  videos)  ERP.  For  this  subject,  there  is  a 
noticeable  overlap  in  the  response  between  600  and  800  ms. 


Figure  22.  ERPs  for  Visual  and  Auditory  Targets  (one  subject).  Top  row:  ERP  for  video  sequences  containing 
targets.  Bottom  row:  ERP  for  target  words. 

To  compare  the  overall  performance  of  the  visual  classifier  with  the  behavioral  response  we  did 
the  following  analysis.  First  we  calculated  the  probability  of  detection  (Pd)  for  the  visual  targets 
based  on  the  behavioral  response  (figure  23a).  We  then  found  the  score  threshold  that  resulted  in 
the  same  Pd  from  the  classifier  (numerical  threshold  values  shown  in  figure  23a  inset).  Using  this 
threshold  we  calculated  the  corresponding  false  alarm  rate  (figure  23b).  For  many  subjects  the 
false  alarm  rate  from  the  classifier  was  nearly  as  good  as  or  even  better  than  the  behavioral  false 
alarm  rate.  In  addition  we  calculated  the  throughput  (clips  per  minute)  for  each  subject  (figure 
23c).  Here  the  behavioral  rate  is  the  true  upper  bound,  denoting  the  average  rate  that  the  subject 
proceeded  through  the  experiment.  The  classifier  throughput  reflects  the  percentage  of  these 
trials  that  were  classified  (i.e.,  free  of  artifacts).  Several  subjects  in  this  study  had  very  few 
artifact  trials  and  therefore  maximal  throughput.  Overall,  this  result  indicates  that  the  current 
state  of  artifact  detection  and  mitigation  is  good,  but  needs  improvement.  A  post  hoc  analysis 
(data  not  shown)  indicated  that  many  of  these  trials  could  have  been  successfully  scored  despite 
the  presence  of  artifacts. 
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Figure  23.  Classifier  Performance  Summary  (all  subjects).  Panel  a:  Probability  of  detection  for  the  classifier  is 

matched  to  behavior.  Panel  b:  False  alarm  rates  for  the  matched  Pd.  Panel  c:  System  throughput  (clips 
per  minute),  reflecting  the  percentage  of  trials  classified  (i.e.,  free  of  artifacts). 

The  preceding  analyses  have  compared  the  classifier  results  with  the  behavioral  response, 
indicating  that,  at  best,  the  classifier  matches  behavioral  performance.  This  result  may  seem  to 
indicate  that  EEG  classification  is  no  better  than  a  behavioral  response.  However,  the  purpose  of 
this  experiment  was  not  to  “outperform”  the  subject.  Rather,  the  purpose  of  this  experiment  was 
to  see  if  a  neural  classification  system  can  continue  to  perform  in  a  multitasking  environment. 
Moreover,  the  experiment  was  not  explicitly  designed  to  be  difficult.  The  rate  of  the  video  clip 
presentation  was  slow  enough  (2  Hz)  for  the  subject  to  respond.  Together,  the  above  results  lay 
the  groundwork  for  a  system  that  could  enhance  overall  performance,  either  by  increasing  RSVP 
presentation  rate  or  allowing  the  subject  to  focus  on  responding  to  the  auditory  (or  other)  task 
alone. 


7.  The  Neural  Correlates  of  Attentional  Impairment 


To  address  this  objective,  the  three  teams  analyzed  their  experimental  data  looking  for  features  in 
the  EEG  signals  that  could  predict  performance  errors,  that  is,  either  incorrect  responses  from 
subjects  or  errors  in  classifying  subjects’  neural  signals.  Finding  reliable,  predictive  features 
could  have  a  major  impact  by  enabling  new  types  of  robust,  adaptive  Soldier-system  interfaces. 

7.1  Study  7:  Predicting  Hits  and  Misses  (ICB) 

In  Study  3B,  a  total  of  eight  subjects  completed  four  separate  session  where  each  session  differed 
in  terms  of  the  probability  that  a  target  was  presented.  In  this  case,  the  target  was  a  face  and  the 
nontargets  were  cars  (both  were  embedded  in  noise).  The  goal  of  Study  7  was  to  analyze  the  data 
to  see  if  a  classifier  could  differentiate  the  trials  in  which  the  subjects  correctly  detected  a  face 
(hits)  from  those  in  which  they  did  not  detect  the  face  (misses).  In  the  initial  analysis,  we  used  an 
LDA-based  classifier  with  temporally  windowed  EEG  amplitude  data.  The  results  are  shown  in 
figure  24.  The  peak  classifier  performance  was  within  the  250-400  ms  time  window  and  the  best 
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classification  occurred  when  the  target  probability  was  0.5.  While  this  may  reflect  an  important 
difference  between  hits  and  misses  as  a  function  of  target  probability,  it  must  be  interpreted  with 
caution  because  there  are  also  more  trials  that  are  included  in  the  classification  analysis  in  that 
condition. 


Classifying  Hits  vs,  Misses 


Time  (ms) 


p(targct) 

—  0.05 

—  0,1 
—  0.25 
— ■  0.5 


Figure  24.  Pattern  classification  performance  for  discriminating 
hit  trials  vs.  miss  trials  plotted  as  a  function  of  time 
and  target  probability. 

We  analyzed  performance  in  more  detail  by  attempting  to  characterize  the  features  of  the  EEG 
signal  that  best  discriminated  between  hits  and  misses.  In  this  analysis,  we  constructed  three 
different  classifiers,  one  that  was  trained  on  the  amplitude  of  the  EEG  data,  one  that  was  trained 
on  power  in  specific  frequency  bands,  and  one  that  was  trained  using  both  amplitude  and  power. 
The  results  of  this  analysis  are  shown  in  figure  25.  As  with  the  analyses  described  above, 
classification  performance  declined  with  increasing  target  probability.  This  was  true  regardless 
of  the  type  of  input.  Performance  was  lowest  when  classification  was  based  on  power  alone.  At 
low  target  probabilities  there  was  a  trend  indicating  that  the  amplitude  only  classifier  was  best, 
but  at  high  target  probabilities  there  was  a  trend  indicating  that  the  amplitude  plus  power  model 
was  the  best.  It  must  be  emphasized  that  these  are  merely  trends  and  not  statistically  significant. 
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Figure  25.  Pattern  classification  accuracy  for  discriminating  hits  from  misses 
when  using  EEG  amplitude  only,  EEG  power  in  alpha,  theta,  beta, 
and  delta,  or  a  combination  of  amplitude  and  power. 

7.2  Study  8:  Predicting  Classifier  Failures  (SAIC) 

In  this  study,  we  analyzed  the  CT2WS  data  looking  at  the  temporal  dynamics  of  the  classifier 
score.  Results  from  Study  4  indicated  that  target  probability  has  a  dramatic  effect  on  classifier 
score.  We  hypothesized  that  whatever  condition  leads  to  mis-classification  (attentional  state, 
temporal  lag,  internal  or  external  noise,  etc.)  would  be  reflected  in  the  temporal  dynamics  of  the 
score.  To  examine  this  we  used  the  subject’s  customized  classifier  model  to  calculate  the  score  of 
each  trial  at  various  time  offsets  (“lags”)  before  and  after  the  presentation  of  the  stimuli 
(±160  ms).  We  then  sorted  these  scores  into  target  and  non-target  trials.  For  target  trials  we 
recorded  the  temporal  offset  between  of  the  peak  (highest)  score  and  stimulus  onset.  Likewise, 
for  non-target  trials  we  recorded  the  temporal  offset  of  the  trough  (lowest)  score  and  stimulus 
onset.  Figure  26a  shows  the  standard  score  distribution  as  a  function  of  this  lag  index  for  both 
target  and  non-target  trials.  Interestingly,  the  closer  the  extreme  score  (peak  or  trough)  is  to  the 
stimulus  onset  the  better  the  classification  accuracy.  To  quantify  this  in  terms  of  performance,  we 
recalculated  the  AUC  for  trials  with  small  absolute  lag  values  (figure  26b).  This  significant 
improvement  was  consistent  over  the  population  (figure  27). 
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Figure  26.  Temporal  Dynamics  and  Classifier  Accuracy  (one  subject).  Panel  a:  Classifier  score  as  a  function  of  lag 
(see  text).  Panel  b:  ROC  curves  for  all  trials  (blue)  and  trials  with  near-zero  lag  values  (green). 


Figure  27.  Classifier  Accuracy  Improvement  (all  subjects). 

AUC  values  for  all  trials  (all  lags)  and  trials  with 
near-zero  lag  values  (best  lags). 

While  taking  into  account  the  temporal  dynamics  of  the  classifier  score  does  lead  to  improved 
accuracy,  it  does  not  identify  the  underlying  reason  for  the  improvement.  Misaligned  trials  (i.e., 
nonzero  lag  index)  could  be  caused  by  some  internal  brain  state  such  as  attentional  or 
preparatory  state  (Kim  et  al.,  2007;  Mathewson  et  al.,  2009).  Alternatively,  they  could  be  caused 
by  some  noise  source  (environmental,  subject  movement,  etc).  However,  the  temporal  dynamics 
in  the  score  are  most  likely  due  to  the  natural  variability  in  the  evoked  (P300)  response.  This 
variability  is  also  reflected  in  the  reaction  time  (see  Study  9).  Further  analysis  of  the  evoked 
response  is  necessary  to  understand  the  source  of  this  phenomenon. 
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7.3  Study  9:  ERP  Latency  and  Reaction  Time  (ARL) 

This  study  focused  on  neurobehavioral  correlations  from  data  acquired  from  12  subjects  during 
an  RSVP  paradigm  used  in  the  CT2WS  program.  The  analysis  focused  specifically  on  the 
relationship  between  target  reaction  time  and  P3  peak  latency.  Many  target  classification 
algorithms  use  features  associated  with  the  P3;  however,  it  is  unclear  what  effect  if  any  the 
latency  of  this  component  may  have  on  classification  accuracy. 

Target  images  were  reviewed  prior  to  experimentation  for  target  detection  difficulty.  Targets  that 
were  partly  occluded  or  difficult  to  detect  were  excluded  from  presentation.  Target  Response 
Times  (RTs)  were  divided  into  quartiles  for  each  subject.  We  focused  on  the  fastest  and  slowest 
performance,  which  correspond  to  the  first  and  last  quartiles  respectively.  Quartiles  were  then 
used  as  epoching  parameters  for  the  EEG  data  in  order  to  create  averaged  ERPs  for  each  subject 
in  each  quartile.  The  top  plot  in  figure  28  shows  target  ERPs  at  electrode  Pz  for  the  first  (red)  and 
last  (blue)  quartiles  as  well  as  the  ERPs  to  non-target  stimuli  (black)  and  the  average  of  the  first 
and  last  quartiles  (green).  While  the  first  and  last  quartiles  have  latency  differences  beginning 
approximately  400  ms  post  target  stimulus,  early  sensory  activation  cannot  account  for  this 
relationship;  performance  quartiles  did  not  change  as  a  function  of  latency  or  amplitude  in  early 
evoked  responses  as  seen  in  the  first  positive  peak  over  electrode  Oz. 


Further  analysis  of  the  RT  and  P3  peak  latency  association  revealed  a  consistent  relationship  at 
the  single  trial  level  as  well  as  across  participants.  Figure  29  shows  an  ERP  image  plot  of  all  first 
and  last  quartile  target  trials.  Trials  were  sorted  by  the  latency  of  target  reaction  time  represented 
by  the  black  line.  Figure  30  depicts  a  significant  positive  correlation  between  RT  and  P3  peak 
latency. 
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Neural  classification  algorithms  focusing  on  P3  amplitude  may  have  increased  performance 
when  P3  latency  information  is  also  considered.  Choosing  the  peak  amplitude  of  the  average 
ERP  response  may  mask  meaningful  performance  related  information  present  in  the  dynamic 
neural  activity. 


O 


Figure  29.  ERP  image  plot  showing  the  P3  amplitude  for  each  target  trial  sorted 
by  reaction  time  performance. 


Figure  30.  Correlation  between  RT  performance  and  P3  peak  latency.  Q1  =  first  quartile, 
Q4  =  last  quartile. 
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8.  Implementation  at  ARL 


This  objective  to  implement  components  of  the  SAIC  real-time  neural  processing  system  at  ARL 
is  the  first  step  along  the  path  to  transition  the  results  of  this  project  to  the  Army.  Work  this  year 
had  two  parts.  First,  SAIC  provided  documentation  and  support  to  ARL  for  their  implementation 
efforts.  Second,  SAIC  defined  a  set  of  metrics  for  assessing  the  performance  of  systems  that 
incorporate  neural  processing.  The  metrics  are  designed  to  support  the  evaluation  of  different 
algorithms  and  implementations.  They  will  be  used  to  measure  improvements  in  system 
performance  as  the  results  of  this  research  are  translated  into  new  methods.  We  begin  with  a 
description  of  the  real-time  system  and  then  discuss  the  metrics. 

8.1  The  SAIC  Real-time  System 

The  SAIC  real-time  EEG  processing  system  (figure  31)  was  developed  under  phase  one  of  the 
DARPA  CT2WS  program.  The  principal  goal  of  this  CT2WS  system  was  to  rapidly  identify  a 
variety  of  targets  over  a  broad  landscape  of  ground-level  imagery  by  combing  the 
complementary  aspects  of  the  computer  and  human  visual  system.  Computer  vision  algorithms 
would  initially  process  the  scene;  identifying  potential  targets  or  threats  as  regions  of  interest 
(ROIs).  These  small  regions  would  then  be  shown  to  the  human  user  under  an  RSVP  paradigm. 
Finally,  the  evoked  neural  response  would  be  used  to  determine  the  presence  or  absences  of  a 
threat. 

The  frontend  or  computer  vision  component  of  this  system  is  not  described  here.  It  is  assumed 
that  the  Experimental  Control  computer  already  has  a  database  of  ROIs.  With  this  assumed 
operation  of  the  system  then  proceeds  as  follows: 

1.  The  user  (or  subject)  is  seated  in  front  of  the  stimulus  display  screen. 

2.  The  EEG  acquisition  system  continuously  acquires  signals  from  the  subject  (128  channels 
at  1  kHz). 

3.  The  subject  initiates  a  session  by  pushing  a  button  (response  pad). 

4.  A  small  set  (approximately  10)  of  ROIs  are  sent  to  from  the  Experimental  Control 
computer  to  the  Stimulus  Presentation  computer  and  shown  to  the  subject.  These  ROIs  are 
time-stamped  using  a  synchronizing  waveform  generated  by  the  EEG  amplifier  (i.e., 
universal  clock).  The  time  stamps  are  also  inserted  into  the  EEG  record  for  synchronization 
of  the  ROI  with  the  corresponding  ERP. 

5.  After  each  ROI  is  presented,  the  Experimental  Control  computer  pulls  the  corresponding 
ERP  from  the  EEG  Acquisition  computer.  The  ERP  is  then  pre-processed  to  check  for 
artifacts  (eye  blinks,  subject  motion,  noise,  etc).  If  the  ERP  is  free  of  artifacts,  a  linear 
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classifier  is  then  used  to  score  the  ROI.  If  the  ERP  contains  artifacts,  it  is  given  a  null 
score. 

6.  ROIs  with  high  positive  (target)  or  negative  (clutter)  scores  are  classified  accordingly. 
ROIs  with  low,  ambiguous  or  null  scores  are  re-cued  for  subsequent  presentation. 

7.  The  Experimental  Control  computer  builds  the  next  set  of  ROIs,  including  any  ROIs  that 
need  to  be  re-shown  (from  step  6).  These  ROIs  are  then  sent  to  the  Stimulus  Presentation 
computer  for  display. 

8.  This  process  (steps  4-7)  continues  until  all  ROIs  are  shown  or  the  subject  terminates  the 
session  via  a  button  press. 

9.  Each  ROI  and  corresponding  score  is  displayed  for  the  operator  on  a  results  screen  that  is 
continuously  updated. 

The  operational  progression  described  above  is  a  high  level  overview  of  the  real-time  system. 
Details  regarding  the  stimulus  content,  presentation  parameters,  classification  algorithm,  and 
scoring  procedure  are  provided  in  a  separate  document,  Documentation  for  the  SAIC  Real-Time 
EEG  System. 
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8.2  Study  10:  Metrics  (SAIC) 

We  defined  a  set  of  metrics  to  be  used  in  assessing  the  performance  of  a  system  that  incorporates 
RSVP  and  neural  processing.  The  purpose  of  the  metrics  is  to  provide  a  quantification  of  the 
improvement,  if  any,  to  a  system  that  includes  a  visual  search  task  such  as  detecting  a  target  or  a 
threat  when  that  task  is  performed  using  neural  processing  and  RSVP.  The  metrics  offer  a  means 
to  compare  performance  against  a  system  that  does  not  use  RSVP.  These  metrics  will  be  applied 
to  provide  an  objective  measure  of  improvement  in  this  project  as  the  prototype  under 
development.  As  part  of  this  study,  we  captured  baseline  metrics  for  the  current  system. 

Three  standard  metrics  will  be  used  to  characterize  performance  in  a  system  that  includes  RSVP: 
probability  of  detection  (Pd),  false  alarm  rate,  and  throughput.  These  metrics  can  be  computed 
for  the  system  with  and  without  RSVP  and  then  compared.  Figure  32  contains  graphs  comparing 
these  metrics  for  the  visual  target  detection  task  implemented  by  SAIC  in  Study  6.  Here  the 
behavioral  data  comes  from  the  subjects  pressing  a  key  when  they  see  a  target  during  the  RSVP 
experiment.  In  this  instance  for  the  behavioral  condition,  throughput  is  expressed  as  the  average 
rate  that  the  subject  progresses  through  the  experiment  in  clips  per  minute.  This  is  the  true  upper 
bound  for  both  the  behavioral  and  automated  system.  The  RSVP  system  throughput  reflects  the 
percentage  of  these  trials  that  were  classified.  The  system  will  not  classify  a  response  if  there  are 
artifacts  such  as  eye  blinks  or  muscle  movement  reflected  in  the  EEG  signal.  Several  subjects  in 
this  study  had  very  few  artifact  trials  and,  therefore,  maximal  throughput.  Overall,  this  result 
indicates  that  the  current  state  of  artifact  detection  and  mitigation  is  good,  but  needs 
improvement.  In  examining  the  trials  that  were  rejected  by  the  system,  we  found  that  many  of 
these  trials  could  have  been  successfully  scored  despite  the  presence  of  artifacts. 


Figure  32.  Classifier  Performance  Summary  (all  subjects).  Panel  a:  Probability  of  detection  for  the  classifier  is 

matched  to  behavior;  Panel  b:  False  alarm  rates  for  the  matched  Pd;  Panel  c:  System  throughput  (clips 
per  minute),  reflecting  the  percentage  of  trials  classified  (i.e.,  free  of  artifacts). 


The  SAIC  real-time  EEG  processing  system  implements  a  linear  classifier  that  applies  a  set  of 
weights  (the  neural  response  model)  to  the  values  coming  from  the  individual  EEG  channels  to 
compute  a  single  score,  which  is  then  used  to  classify  the  response.  Thus  the  classifier  produces  a 
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score  for  each  stimulus  based  on  the  neural  response  of  the  subject.  A  set  of  metrics  for  the 
RSVP  system  can  be  computed  based  on  an  analysis  of  the  classifier  output.  One  such  metric  is 
the  area  under  the  ROC  curve  (AUC)  which  is  standard  for  characterizing  how  well  an 
automated  algorithm  performs,  where  AUC=1  indicates  a  perfect  system  and  AUC=0.5  is  a 
system  performing  at  the  level  of  chance.  Figure  33  shows  a  single  ROC  curve  for  one  subject 
from  Study  6.  Figure  34  shows  a  chart  with  the  AUC  for  all  subjects. 


Figure  33.  ROC  curve  for  a  single  subject  for  classifier  scores 
generated  using  a  general  neural  response  model 
(in  blue)  and  for  scores  generated  by  using  a  model 
built  for  this  individual  (in  green). 


Subject  Number 


Figure  34.  Bar  chart  showing  the  area  under  the  ROC  curve  (AUC) 
for  all  subjects  in  Study  6. 
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The  real-time  system  uses  a  threshold  of  zero  for  the  classifier  scores:  a  score  greater  than  zero 
indicates  a  target  and  less  than  zero  indicates  a  non-target.  A  separate  analysis  can  be  carried  out 
to  compare  the  classifier  results  with  the  behavioral  results  by  adjusting  this  threshold.  For  each 
subject,  suppose  we  set  the  classifier  threshold  to  the  largest  number  t  such  that  the  probability  of 
detection  for  the  threshold  t  is  equal  to  the  subject’s  behavioral  probability  of  detection.  Figure 
35a  shows  a  histogram  of  this  behavioral-matched  threshold  for  the  subjects  in  Study  6.  Using 
this  threshold,  we  can  then  calculate  the  corresponding  false  alarm  rate  and  compare  it  to  the 
false  alarm  rate  for  behavior.  Figure  35b  shows  that  for  many  subjects,  the  false  alarm  rate  from 
the  classifier  is  almost  as  good  as,  and,  in  some  cases,  even  better  than  the  behavioral  false  alarm 
rate. 


Threshold 


Behavioral 


Figure  35.  Metrics  for  classifier  performance.  Panel  c:  Histogram  of  the  classifier  score  thresholds  the  give  the  same 
Pd  as  behavioral  detection  for  each  subject;  Panel  b:  matched  Pd  for  behavioral  and  classifier 
performance;  Panel  c:  plot  of  false  alarm  rate  for  behavioral  versus  classifier  with  matched  Pd. 

If  the  RSVP  task  is  the  first  step  of  an  image  triage  process  (Sajda  et  ah,  2003;  Sajda  et  al., 

2010),  we  define  additional  metrics  to  characterize  performance.  In  this  case,  the  button  presses 
or  EEG  classifier  scores  from  the  RSVP  component  can  be  used  to  sort  the  video  clips  for 
subsequent  review  by  a  human  operator.  For  the  behavioral-based  system,  we  can  re-order  the 
clips  so  that  the  clips  that  elicited  a  button  press  are  all  at  the  top  of  the  queue.  For  the  RSVP- 
based  system,  the  clips  are  re-ordered  based  upon  score,  from  the  highest  score  to  the  lowest. 

The  resequencing  based  on  behavioral  and  RSVP  input  can  be  compared  with  the  original 
ordering  by  examining  the  accuracy  in  the  following  sense.  In  figure  36,  the  re-sequencing  of  an 
entire  session  for  two  subjects  is  presented.  Here,  the  fraction  of  targets  is  plotted  as  a  function  of 
sequence  index  for  the  2800  video  clips.  The  resequencing  via  button  press  or  classifier  score  is 
bounded  on  the  lower  end  by  the  original  sequence,  where  targets  and  non-targets  are  presented 
randomly,  and  the  on  the  upper  end  by  optimal  resequencing,  where  all  targets  are  presented 
before  the  non-targets.  For  the  two  subjects  shown  in  figure  36,  the  button  press  initially 
outperforms  the  classifier,  that  is,  more  targets  appear  at  the  front  of  the  sequence.  After  a  certain 
percentage  of  the  targets  appear,  the  classifier  results  begin  to  outperform  the  button  press.  For 
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the  last  10-25%  of  the  targets  (for  these  two  subjects),  more  appear  at  the  front  of  the  classifier- 
ordered  sequence. 


Subject  100 


Subject  102 


Sequence  index 


Figure  36.  Re-sequencing  accuracy.  Fraction  of  target  images  presented  as  a  function  of  sequence  index.  Lower 

bound:  original  sequence  where  targets  where  presented  randomly.  Upper  bound:  optimal  re-sequencing 
where  all  targets  are  presented  before  any  non-targets. 

The  analysis  of  resequencing  accuracy  compares  the  performance  of  the  classifier  to  a  subject’s 
button  presses.  Often  the  RSVP-based  triage  and  review  will  replace  a  totally  manual  process  of 
looking  for  targets  while  stepping  through  a  sequence  of  images.  A  metric  that  is  useful  for 
comparisons  is  the  number  of  targets  detected  per  unit  time.  If  the  goal  of  the  process  is  to  find 
targets  quickly,  the  best  system  is  the  one  that  results  in  finding  the  most  targets  soonest.  Figure 
37  illustrates  a  useful  method  for  comparing  performance.  For  the  baseline  condition,  the 
subjects  manually  review  all  video  clips  looking  for  targets.  We  assume  they  view  the  clips  for 
500  ms  each  and  have  an  average  response  time  of  750  ms.  Assuming  the  targets  are  evenly 
distributed  through  the  original  sequence,  the  targets  per  time  graph  for  the  baseline  is  a  straight 
line.  With  RSVP-based  triage,  first  the  subject  performs  RSVP;  the  images  are  re-sequenced 
based  upon  the  classifier  score,  and  the  subject  reviews  the  results.  In  figure  37,  the  dashed 
vertical  line  represents  the  end  of  the  RSVP  period  and  the  red  line  shows  the  targets  found 
plotted  against  time.  The  area  between  the  red  curve  and  the  baseline  (positive  if  the  red  line  is 
above  the  baseline  graph)  quantifies  the  benefit  of  triage.  In  this  case,  the  benefit  is  modest.  If  we 
assume  a  presentation  rate  of  100  ms/image  clip,  the  results  are  more  substantial  (figure  37b). 
This  metric  can  help  define  the  parameters  for  RSVP  and  inform  how  an  RSVP  component 
might  be  utilized  in  the  crew-station. 
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Figure  37.  Image  triage  performance.  Number  of  targets  detected  as  a  function  of  time  for  baseline  and  triage 
(RS  VP-sorted)  condition.  Dashed  line  indicates  end  of  RS  VP  component  and  beginning  of  manual 
review.  Panel  a:  Plot  shows  the  performance  in  the  current  500  ms  per  clip  condition;  Panel  b:  plot 
shows  the  performance  for  a  hypothetical  1 00  ms  per  clip  condition  (assuming  similar  classifier 
performance). 


9.  Summary  of  Results 


We  summarize  the  key  results  from  Year  1  in  table  4. 
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Table  4.  Summary  of  the  results  by  study  number. 


Result 

Study 

When  RSVP  is  presented  at  fixation,  there  is  no  performance  impact  due  to  RSVP  rate. 

1 

Performance  degrades  directly  as  a  function  of  the  distance  of  the  image  from  fixation. 

1 

For  a  given  level  of  performance,  targets  presented  in  the  periphery  must  be  more  salient  than 
targets  presented  at  fixation. 

2 

Behavioral  performance  degrades  as  target  frequency  increases. 

3 

Pattern  classification  performance  peaks  at  300  ms  post-stimulus  and  is  highest  when  the  target 
frequency  is  0.1  or  0.25. 

3 

If  a  classifier  is  trained  on  data  from  one  target  frequency  and  then  applied  to  data  from  another 
target  frequency,  the  performance  is  degraded.  Training  with  data  from  a  higher  target 
probability  condition  results  in  the  least  degradation. 

3 

Classifier  scores  are  inversely  correlated  with  target  frequency. 

4 

Target  frequency  can  be  estimated  from  the  distribution  of  scores  and  used  to  compensate  for 
changes  in  this  parameter. 

4 

Behavioral  performance  is  similar  under  single  and  dual  task  conditions. 

5 

Pattern  classification  performance  for  the  visual  task  peaks  at  300  ms  post-stimulus  and  is 
unaffected  by  the  second  task. 

5 

Classifier  performs  poorly  (at  chance)  on  visual  stimuli  that  are  shown  while  the  subject  is 
carrying  out  the  auditory  task  only. 

5 

There  is  no  significant  difference  in  behavioral  performance  for  the  visual  task  alone  compared 
with  the  visual  and  auditory  tasks  together. 

6 

There  is  no  significant  difference  in  classifier  performance  for  the  visual  task  alone  compared 
with  the  visual  and  auditory  tasks  together  (for  the  subjects  with  high  classifier  performance). 

6 

Classifier  performance  degrades  for  combined  auditory  and  visual  tasks  for  subjects  with  poor 
classifier  models. 

6 

Result 

Study 

For  many  subjects,  separate  classifiers  could  detect  visual  and  auditory  targets. 

6 

The  pattern  classifier  can  discriminate  between  successfully  detect  targets  and  those  that  are 
missed  based  on  neural  activity. 

7 

Classification  performance  is  higher  when  all  post- stimulus  activity  is  used  at  once  (right  panel). 

7 

Although  classifiers  using  either  amplitude  or  power  alone  perform  above  chance,  amplitude  is 
better.  There  is  little  or  no  benefit  when  including  both  amplitude  and  power  together. 

7 

Latency  in  the  ERP  and  the  classifier  score  predict  classifier  accuracy  and  can  be  used  to 
improve  system  performance. 

8 

Latency  in  the  ERP  mirrors  the  variably  in  the  behavioral  response  (RT).  This  variability  is 
partially  a  reflection  of  the  attentional  state  of  the  individual. 

9 

S  AIC’s  current  real-time  system  can  achieve  performance  levels  nearing  those  of  behavior  (the 
upper-bound  in  this  metric  formulation). 

10 

Improvements  can  be  made  through  1)  better  classifier  models,  2)  Incorporation  of  attentional 
state  and  temporal  dynamics  into  classification  scheme  and  3)  enhanced  EEG  artifact  detection 
and  mitigation. 

10 
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ACC 

ARL/HRED 

ATO 

AUC 

BCIT 

CT2WS 

DARPA 

EEG 

ERP 

fMRI 

FPR 

HD-Cog 

IA 

ICB 

LDA 

MGV 

NIA 

NTG 

P3 

Pd 

ROC 

ROI 

RSVP 

RT 

SAIC 

SOA 

TARDEC 

TPR 

TRG 


Attentional  Blink 
accuracy 

Army  Research  Laboratory/Human  Research  and  Engineering  Directorate 

Army  Technology  Objective 

Area-Under-the-Curve 

Brain-computer  Interactive  Technologies 

Cognitive  Technology  Threat  Warning  System 

Defense  Advanced  Research  Projects  Agency 

Electroencephalography 

Event-related  Potential 

Functional  Magnetic  Resonance  Imaging 

False  Positive  Rate 

High-Definition  Cognition  in  Operational  Environments 
Imagery  Analyst 

Institute  for  Collaborative  Biotechnologies 

linear  discriminant  analysis 

manned  ground  vehicles 

Neurotechnology  for  Intelligence  Analysts 

non-target 

P300 

probability  of  detection 

Receiver  Operating  Characteristic 

region  of  interest 

Rapid  Serial  Visual  Presentation 

Reaction  Time,  Response  Time 

Science  Applications  International  Corporation 

stimulus  onset  asynchrony 

Tank  Automotive  Research  Development  and  Engineering  Command 

True  Positive  Rate 

target 
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